Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](nereids)fix rangeSet intersect in PartitionPredicateToRange #46087

Merged
merged 2 commits into from
Jan 3, 2025
Merged

[fix](nereids)fix rangeSet intersect in PartitionPredicateToRange #46087

merged 2 commits into from
Jan 3, 2025

Conversation

feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Dec 27, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #44586

Problem Summary:
The intersection operation of RangeSet is incorrect and will incorrectly calculate the empty set. This PR fixes this problem.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow requested a review from 924060929 December 27, 2024 04:24
@morrySnow
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32671 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 16b75c61e8940fdba3014a9fdf397348ed3f06d4, data reload: false

------ Round 1 ----------------------------------
q1	17629	6204	6086	6086
q2	2049	302	168	168
q3	10413	1247	725	725
q4	10200	868	427	427
q5	7517	2188	1980	1980
q6	216	180	146	146
q7	905	739	635	635
q8	9238	1405	1182	1182
q9	5238	4970	4917	4917
q10	6770	2318	1898	1898
q11	479	286	264	264
q12	357	370	221	221
q13	17757	3775	3005	3005
q14	238	238	220	220
q15	565	495	485	485
q16	649	618	590	590
q17	574	861	336	336
q18	7250	6542	6345	6345
q19	1247	998	541	541
q20	322	337	190	190
q21	2858	2162	1990	1990
q22	376	330	320	320
Total cold run time: 102847 ms
Total hot run time: 32671 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6257	6266	6307	6266
q2	243	341	231	231
q3	2245	2684	2435	2435
q4	1470	1872	1427	1427
q5	4380	4775	4875	4775
q6	220	188	144	144
q7	2151	1992	1848	1848
q8	2628	2833	2679	2679
q9	7324	7258	7305	7258
q10	3073	3364	2855	2855
q11	586	511	505	505
q12	684	802	622	622
q13	3545	3854	3088	3088
q14	282	297	278	278
q15	579	515	510	510
q16	663	691	642	642
q17	1209	1739	1248	1248
q18	7844	7402	7080	7080
q19	806	1108	1073	1073
q20	1921	2006	1816	1816
q21	5430	5153	4938	4938
q22	644	652	602	602
Total cold run time: 54184 ms
Total hot run time: 52320 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16b75c61e8940fdba3014a9fdf397348ed3f06d4, data reload: false

query1	994	380	386	380
query2	6529	2365	2387	2365
query3	6715	218	212	212
query4	33703	23936	23859	23859
query5	4364	645	467	467
query6	314	214	208	208
query7	4633	505	312	312
query8	318	271	240	240
query9	9523	2834	2802	2802
query10	448	324	257	257
query11	18120	15444	15283	15283
query12	155	107	103	103
query13	1651	579	427	427
query14	9682	6993	6921	6921
query15	259	188	192	188
query16	8364	569	406	406
query17	1569	734	557	557
query18	2091	390	294	294
query19	210	179	150	150
query20	121	121	117	117
query21	210	120	102	102
query22	4342	4318	4393	4318
query23	35090	33390	33795	33390
query24	7500	2337	2323	2323
query25	485	456	399	399
query26	1192	275	156	156
query27	2691	460	340	340
query28	5878	2488	2455	2455
query29	753	537	419	419
query30	226	183	151	151
query31	1000	907	826	826
query32	94	63	60	60
query33	504	356	299	299
query34	773	854	535	535
query35	804	814	739	739
query36	1016	1079	962	962
query37	124	111	82	82
query38	4135	4357	4305	4305
query39	1504	1431	1433	1431
query40	211	114	103	103
query41	47	48	67	48
query42	125	106	104	104
query43	543	548	509	509
query44	1326	803	809	803
query45	180	174	172	172
query46	942	1058	671	671
query47	1899	1915	1864	1864
query48	400	425	330	330
query49	777	481	412	412
query50	643	672	419	419
query51	7147	7261	7134	7134
query52	107	106	98	98
query53	234	273	190	190
query54	487	510	414	414
query55	85	81	80	80
query56	252	282	246	246
query57	1234	1174	1101	1101
query58	243	243	236	236
query59	3215	3102	3364	3102
query60	291	311	282	282
query61	134	131	130	130
query62	907	801	728	728
query63	236	198	194	194
query64	4553	994	657	657
query65	3299	3200	3241	3200
query66	1072	428	315	315
query67	15966	15849	15687	15687
query68	7864	784	510	510
query69	464	296	312	296
query70	1238	1185	1172	1172
query71	442	293	255	255
query72	5818	3784	3878	3784
query73	673	763	362	362
query74	10674	9254	8918	8918
query75	4149	3156	2707	2707
query76	3735	1192	758	758
query77	784	381	288	288
query78	10137	10186	9665	9665
query79	2488	913	611	611
query80	607	530	431	431
query81	503	268	239	239
query82	430	153	127	127
query83	165	161	144	144
query84	242	97	76	76
query85	836	351	296	296
query86	408	289	304	289
query87	4440	4420	4632	4420
query88	4854	2227	2205	2205
query89	393	351	296	296
query90	1849	195	192	192
query91	141	140	104	104
query92	63	61	54	54
query93	1589	906	528	528
query94	654	376	301	301
query95	345	255	255	255
query96	495	613	283	283
query97	2793	2824	2655	2655
query98	232	210	197	197
query99	1681	1521	1444	1444
Total cold run time: 295825 ms
Total hot run time: 191931 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16b75c61e8940fdba3014a9fdf397348ed3f06d4, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.60	0.10	0.10
query5	0.44	0.42	0.42
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.55	0.56	0.56
query11	0.16	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.62	0.60
query14	2.74	2.83	2.75
query15	0.90	0.83	0.83
query16	0.42	0.39	0.38
query17	1.03	1.00	1.03
query18	0.23	0.22	0.20
query19	1.96	2.03	1.87
query20	0.02	0.01	0.01
query21	15.36	0.97	0.59
query22	0.78	0.84	0.77
query23	15.06	1.44	0.58
query24	3.03	1.04	1.97
query25	0.14	0.23	0.13
query26	0.23	0.14	0.14
query27	0.05	0.07	0.06
query28	14.33	1.52	1.05
query29	12.57	3.91	3.26
query30	0.26	0.09	0.07
query31	2.82	0.60	0.41
query32	3.23	0.54	0.46
query33	3.07	3.20	3.08
query34	16.79	5.09	4.48
query35	4.48	4.46	4.49
query36	0.65	0.52	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.3 s
Total hot run time: 31.64 s

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9fe55e9ad342857634cbc016d1ae99d1ce64301d, data reload: false

------ Round 1 ----------------------------------
q1	17601	6114	6058	6058
q2	2049	297	180	180
q3	10579	1271	745	745
q4	10317	876	447	447
q5	8296	2210	1977	1977
q6	202	182	149	149
q7	885	741	607	607
q8	9241	1354	1199	1199
q9	5206	4875	4919	4875
q10	6773	2317	1856	1856
q11	504	290	269	269
q12	356	362	232	232
q13	17774	3593	3029	3029
q14	233	226	208	208
q15	558	521	496	496
q16	638	624	588	588
q17	574	860	326	326
q18	7067	6563	6330	6330
q19	1982	971	565	565
q20	319	328	195	195
q21	2878	2161	2043	2043
q22	369	335	316	316
Total cold run time: 104401 ms
Total hot run time: 32690 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6294	6188	6233	6188
q2	237	327	234	234
q3	2208	2624	2297	2297
q4	1478	1877	1384	1384
q5	4337	4729	4675	4675
q6	178	171	138	138
q7	1964	1847	1695	1695
q8	2547	2683	2619	2619
q9	6966	6882	6887	6882
q10	2962	3230	2714	2714
q11	557	506	498	498
q12	685	733	572	572
q13	3252	3627	3013	3013
q14	274	293	262	262
q15	561	510	512	510
q16	645	680	650	650
q17	1202	1688	1231	1231
q18	7291	7092	7036	7036
q19	848	1155	1072	1072
q20	1929	1955	1769	1769
q21	5538	5029	4904	4904
q22	626	623	574	574
Total cold run time: 52579 ms
Total hot run time: 50917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191686 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9fe55e9ad342857634cbc016d1ae99d1ce64301d, data reload: false

query1	992	388	364	364
query2	6502	2514	2633	2514
query3	6703	222	218	218
query4	33915	23691	23356	23356
query5	4354	647	466	466
query6	289	191	184	184
query7	4629	517	304	304
query8	289	243	233	233
query9	9799	2752	2726	2726
query10	482	310	241	241
query11	18095	15375	15078	15078
query12	151	106	101	101
query13	1647	531	398	398
query14	10517	7466	7610	7466
query15	255	187	189	187
query16	8064	606	441	441
query17	1557	742	552	552
query18	2084	392	299	299
query19	223	193	178	178
query20	117	113	108	108
query21	211	125	103	103
query22	4393	4345	4374	4345
query23	33983	33383	33473	33383
query24	6614	2380	2232	2232
query25	471	454	367	367
query26	1188	280	163	163
query27	1995	471	338	338
query28	5317	2489	2449	2449
query29	711	536	414	414
query30	231	186	160	160
query31	1003	899	830	830
query32	89	63	61	61
query33	524	360	300	300
query34	793	856	524	524
query35	818	838	752	752
query36	1023	1052	954	954
query37	122	100	78	78
query38	4139	4165	4071	4071
query39	1504	1459	1518	1459
query40	215	124	109	109
query41	51	49	48	48
query42	126	112	103	103
query43	519	545	501	501
query44	1327	808	808	808
query45	187	180	169	169
query46	926	1078	649	649
query47	1881	1933	1829	1829
query48	408	418	350	350
query49	778	467	389	389
query50	648	665	388	388
query51	7292	7106	7005	7005
query52	102	100	97	97
query53	226	272	188	188
query54	472	484	402	402
query55	84	80	77	77
query56	257	270	234	234
query57	1217	1164	1127	1127
query58	234	231	227	227
query59	3099	3281	3029	3029
query60	275	260	254	254
query61	110	108	105	105
query62	866	820	752	752
query63	230	195	196	195
query64	4292	981	651	651
query65	3230	3175	3213	3175
query66	1049	415	321	321
query67	15983	15715	15559	15559
query68	9161	761	502	502
query69	493	288	262	262
query70	1241	1174	1135	1135
query71	438	292	255	255
query72	5908	3776	3890	3776
query73	667	786	377	377
query74	10228	9073	8904	8904
query75	4638	3174	2696	2696
query76	5243	1194	790	790
query77	982	363	273	273
query78	9917	10308	10892	10308
query79	1600	819	616	616
query80	607	545	437	437
query81	496	275	224	224
query82	272	154	121	121
query83	173	167	151	151
query84	235	87	77	77
query85	793	347	335	335
query86	352	321	302	302
query87	4514	4449	4537	4449
query88	3405	2248	2216	2216
query89	402	324	299	299
query90	1885	192	191	191
query91	133	138	116	116
query92	68	57	58	57
query93	989	754	528	528
query94	660	398	299	299
query95	329	270	258	258
query96	484	611	281	281
query97	2732	2809	2655	2655
query98	217	207	203	203
query99	1882	1541	1455	1455
Total cold run time: 293021 ms
Total hot run time: 191686 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9fe55e9ad342857634cbc016d1ae99d1ce64301d, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.10
query5	0.43	0.39	0.42
query6	1.14	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.55	0.59	0.56
query11	0.14	0.10	0.10
query12	0.13	0.12	0.12
query13	0.61	0.61	0.60
query14	2.81	2.75	2.75
query15	0.90	0.82	0.83
query16	0.38	0.38	0.38
query17	1.04	1.00	1.06
query18	0.22	0.22	0.22
query19	1.85	1.73	1.90
query20	0.01	0.02	0.01
query21	15.36	0.96	0.59
query22	0.77	0.98	0.67
query23	15.00	1.34	0.55
query24	2.59	1.21	1.49
query25	0.13	0.18	0.10
query26	0.29	0.16	0.15
query27	0.04	0.04	0.04
query28	13.78	1.55	1.05
query29	12.58	3.98	3.26
query30	0.25	0.09	0.06
query31	2.83	0.61	0.39
query32	3.21	0.55	0.46
query33	3.09	3.10	3.08
query34	16.61	5.11	4.49
query35	4.54	4.54	4.52
query36	0.63	0.49	0.48
query37	0.09	0.05	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.02 s
Total hot run time: 31.52 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 2, 2025

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 7b45bc3 into apache:master Jan 3, 2025
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@924060929 924060929 924060929 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@feiniaofeiafei @hello-stephen @morrySnow @doris-robot @924060929